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Abstract 

This paper develops a controller synthesis method for distributed LQG control problems under 
output-feedback. We consider a system consisting of three interconnected linear subsystems with a 
delayed information sharing structure. While the state-feedback case has previously been solved, the 
extension to output-feedback is nontrivial as the classical separation principle fails. To find the optimal 
solution, the controller is decomposed into two independent components: a centralized LQG-optimal 
controller under delayed state observations, and a sum of correction terms based on additional local 
information available to decision makers. Explicit discrete-time equations are derived whose solutions 
are the gains of the optimal controller 



I. Introduction 

Control with information constraints imposed on decision makers, sometimes called team 
theory or distributed control, has been very challenging for decision theory researchers. In 
general, several classes of these problems are currently computationally intractable [[T|. Early 
work [2| showed that even in a simple static linear quadratic decision problem, complex nonlinear 
decisions could outperform any given linear decision. As a result, much research has focused 
on identifying classes of decentralized control problems that are tractable [[3|-[|5|. 

Distributed Linear Quadratic Gaussian (LQG) control with communication delays has a rich 
literature dating back to the 1970s. Even though the LQG problem under one-step delay in- 
formation sharing pattern has been solved in [|6|-[|9j, generalizing their approaches to other 



delay structures is non-trivial. In [10| and |11|, a computationally efficient solution for the 
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LQG output-feedback problem with communication delays was presented using a state space 
formulation and covariance constraints, but the controller structure is not apparent from the 
corresponding semi-definite programming solution. In [12], the authors consider LQG control 
with communication delays for the three interconnected systems. While they provide an explicit 
solution, their approach is restricted to state-feedback and assumes independence of disturbances 
acting on each subsystem. 

In this paper, we generalize the results in (TI] to output-feedback and correlated disturbances. 
We consider three interconnected systems over a strongly connected graph, which implies infor- 
mation from neighbors is available with one step delay and the global information is available to 
all decision makers with two step delay. We derive an output-feedback law that minimizes a finite- 
horizon quadratic cost. The problem considered here provides the fundamental understanding 
for general delay structures. 

The main contribution of this paper is the explicit state-space realization of the LQG output- 
feedback problems with communication delays. The problem is solved by decomposing the 
controller into two components. One is the same as centralized LQG problem under two-step 
information delay and the other is the sum of correction terms based on local information 
available to decision makers. Specifically, the optimal control has the form 

u{k) = F{k){y{k) - CxW(A;)) + F^^\k){y{k - 1) - Cx{k - l\k - 2)) + L{k)x{k\k - 2), 

where x{k — l\k — 2) and x{k\k — 2) is the one- and two-step estimation of the state based on the 
common two-step delayed information, and x^^^{k) is an improved state estimate based on local 
information up to time k—1 available to decision makers at time k. While the gain matrix L might 
be full (in fact, it is the standard LQR gain computed via discrete-time Riccati recursion), the 
gain matrices F and F^^^ have a spars ity structure that complies with the information constraints. 
We further show that F and F^^l can be computed via convex programming. 

The paper is organized as follows. Section In] defines the general problem studied in this 



paper. In Section III we review the standard discrete time Kalman filter and derive an optimal 
estimation algorithm for the three-player problem. In Section |IV] it is shown that the three-player 
control problem can be separated into two optimization problems. The main result of this paper 



is stated in Section |Vj Numerical results are given in Section |VI] and finally conclusions and 
future work are outlined in Section IVIII 
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A. Notation 



Throughout the paper, we use the following notation: matrices are written in uppercase letters 
and vectors in lowercase letters. The sequence x{0), x{l), . . . , x{k) is denoted by x{0 : k). The 
symbol / denotes the identity matrix whose size can be determined from its context. For a matrix 
X partitioned into blocks, [^JsiSj denotes the sub-matrix of X containing exactly those rows and 



columns corresponding to the sets Si and 5'2, respectively. For instance [X]{i}{2.3} 

The trace of a square matrix X is denoted by Tr{X}. Given A G W" 



Xi2 



terms of its columns as A 
vector 



we can write A in 



Then operation vtc{A) results in an mn x 1 column 



vec(A) 



ai 



For A e R""'"'* and S e R'■^^ the operation A<S)B e R'^'-x"^ denotes the Kronecker product 
of A and B. We denote the expectation of a random variable x by E{x}. The conditional 
expectation of x given y is denoted by E{,t|?/}. The covariance of zero-mean random vectors x 
and y, defined by E{xy^}, is denoted by Co\{x,y}. 



II. Problem Formulation 
Consider the following linear discrete time system composed of m interconnected subsystems 

m 

Xi{k + 1) = '^AijXj{k) + BiUi{k) + Wi{k) 

i=i (1) 

yi{k) = CiXi{k) + Vi{k), 

for i = 1, . . . ,m. Here, Xi e R"* is the state , Ui e R^' is the control signal, yi e R^* is the 
measurement output, Wi is the disturbance, and Vi is the measurement noise of subsystem i. 
Here, Aj, e ]R"i><"j , 5- e j^^ix-ji and Cj e RPiX'*^ are constant matrices. Let us define 





Xi 




Ui 




yi 




Wi 




Vl 


X — 




, U = 




, y = 




, w = 




, V = 














ym 











4 



Then the system dynamics ([T]) can be written as 

x{k + 1) = Ax{k) + Bu{k) + w{k) 
y{k) = Cx{k) + v{k), 



(2) 



where A = [A, 



B = dmg{B^,...,Brn) G 



onxq 



and C = diag(Ci, . . . , C^) e M^^ 



Both IV and are assumed to be Gaussian white noises with covariance matrix 

-1 



E 



w{k) 
v{k) 



w{l) 
vil) 



5{k-l) 



W 
V 



where 5{k - I) = I if k = I and 5{k-l)=Qiik^ I. 

Assumption 1 V is positive definite. 

The interconnection structure of system (|2]) can be represented by a graph Q whose nodes 
correspond to subsystems. The graph Q has an arrow from node j to node i if and only if 
Aij 7^ {i.e. if Xj{k) influences Xi{k + 1)). Assume that Q is strongly connected and passing 
information from one node to another along the graph takes one time step. Let dij be the length 
of the shortest path from node i to node j with da = Then node i receives the information 
available to node j after dji time steps, and hence the available information set of subsystem i 
at time k is given by 



Xi{k) = {yi{0 : k - du), . . . ,yi{0 : k), ... ,2/^(0 : k - dmi)}. 



(3) 



The control problem is to minimize finite-horizon cost 

T r 

x{k) 



N-1 



^ = E E 



A:=0 



x{k) 
u{k) 



Q 



u{k) 



+ x{NYQox{N) 



(4) 



subject to inputs of the form 



Ui{k) = fii{li{k)), i = l,...,m, 



where yU, is the Borel-measurable function. Matrix Q is partitioned according to the dimensions 
of X and u as 

Qxx Qi 
Qxu Qi. 



Q 
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Assumption 2 The matrices Qq and Q are positive semi-definite, and Quu positive definite. 

The information structure (|3]) can be viewed as the consequence of delays in the communication 
channels between the controllers. The assumptions about the information structure and the 
sparsity of dynamics guarantee that information propagates at least as fast as the dynamics 
on the graph. This information pattern is a simple case of partially nested information structure 
that has been studied in [j3J. The optimal controller with this information pattern exists and it is 
unique and linear. 

While the approach proposed in this paper applies for linear systems over strongly connected 
graphs, we will concentrate on a simple delayed information control problem referred to as the 
three-player problem shown in Figure [T| For this problem, the system matrices have the structure 





An 





Ai3 






















A = 


A21 


A22 





, B = 





B2 





, c = 





C2 










A32 


^33 










^3 













and the information available to each player at time k is 

Ii{k) = {yi{k),y,{k - l),y3{k - 1), y(0 : k - 2)}, 

Mk) = {y2{k),yi{k - l),y2{k - 1), 1/(0 : k - 2)}, 

Uk) = {y,{k),y2{k - l),y3{k - l),y(0 : k - 2)}. 

Since the information structure is partially nested, the optimal controller of each player is a 
linear function of the elements of its information set. Hence, 

ui{k) = hi{yiik)) + fu{yiik - l),ys{k - 1)) + /i3(y(0 : - 2)), 

U2{k) = f2i{y2{k)) + f22{yi{k - l),y2{k - 1)) + /23(y(0 : - 2)), 

usik) = /3i(z/3(A:)) + f32{y2{k - l),ys{k - 1)) + fs^iyiO : A; - 2)), 

where fij is a linear function for all i, j. Therefore, u{k) can be expressed as 

uik) = F{k)y{k) + G{k)y{k - 1) + /(|/(0 : A; - 2)), (5) 

where 





/l3 




Fii{k) 




Gii{k) 





Gisik) 


/ = 


/23 


, F{k) = 


F22{k) 


, G{k) = 


G2iik) 


G22{k) 







/33 




F33(A;) 







G32ik) 


Gs3ik) 
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Fig. 1. The graph illustrates the interconnection structure of three players. The state of Player 1 at time fc + 1 depends directly 
on the state of Player 3 at time k since Ai^ ^ 0, hence there is an arc from node 3 to node 1 in the interconnection graph. 
On the other hand, since A12 = 0, Player 1 is not affected directly by the state of Player 2, and there is no arc from node 2 to 
node 1 in the interconnection graph. 

Note that the sparsity structures of F and G comply with the information constraints at time 
k and k — 1, respectively. The control problem is now to find matrices F and G, as well as a 
hnear function /, that minimize J. 



III. Estimation Structure 
This section presents an optimal estimation algorithm for the three-player problem. First, we 



provide a short summary of standard Kalman filtering in Subsection III-A[ Next, Subsection III-B 



sketches a derivation of the estimation algorithm. Finally, some properties of the algorithm are 



given in Subsection III-C 



A. Preliminaries on Standard Kalman Filtering 

Consider a linear system on the form whose initial state x(0) is Gaussian with zero mean 
and covariance matrix Pq. Let us define the following variables 

x{k\k-l) := E{x{k)\y{0 : k - 1)} 
e{k) := x{k) — x{k\k — 1) 
P{k) := E{e(A:)e^(A;)}. 

Here, x{k\k — 1) is the one-step prediction of the state, e{k) is the prediction error, and P{k) 
is the covariance matrix of the prediction error at time k. Assume that u(k) is a deterministic 



(6) 



function of y(0 : k). The Kalman filter equations can be written as follows ( [13|) 

x{k + = Ax{k\k - 1) + Bu{k) + K{k) [y{k) - Cx{k\k - 1)) 
P{k + 1) = AP{k)A^ + W - AP{k)C^{CP{k)C^ + vy^CP{k)A^, 
with x(0| — 1) = and P(0) = Pq. Here, K{k:) is the optimal Kalman gain given by 

K{k) = AP{k)C^ {CP{k)C^ + yy^ ■ 
The innovations are defined by 

m=yik)-Cx{k\k-l). (7) 

The following proposition will be useful when deriving the optimal estimation algorithm for the 
three-player problem. 



Proposition 1 ( p3y) The following facts hold: 

(a) E{x{k)y{kf} = P{k)C^. 

(b) y{k) is an uncorrelated Gaussian process with covariance matrix Y{k) = CP{k)C'^ + V. 
Moreover, under Assumption 1, Y{k) is positive definite. 

(c) y{k) is independent of past measurements 

E{y(%^(j)} = forj<k. 



B. Kalman Filtering for Three-player Problem 

Let 'lf\k) be the set of all measurements up to time step k — 1 that are available to Player i 
at time k. For example, 

Xl\k) = {y,{k - l),y,{k - 1), y(0 : A; - 2)} . 

It is easy to verify that xf\k) C y(0 : /c — 1), i.e. it does not have access to all measurements 
taken at time k — 1. Hence, players cannot execute the one-step prediction of the standard Kalman 
filter Xi{k\k — 1) dX time k. Define 

xf{k):=E[x,{k)\xf\k)], 2 = 1,2,3. 

We will now derive explicit expressions for these quantities. 



g 



Note that y{0 : k — 2) is the piece of information available to all players. Thus, x{k — l\k — 2) 
can be computed by each player at time k. To see how the optimal estimation algorithm for the 
three-player problem is derived, consider Player 1. Let [A]i denote the ith block row of A. Then, 

x?{k) =E\^xi{k)\lP{k)} 

= [A],E [xik - l)\I?ik)] + B^E [u,{k - l)\I?{k)] 

= [A^E {x{k - l)\y^{k - l),y,{k - l),y{0 : k - 2)} + B,u,{k - 1), 

where we used the independence of Wi{k — 1) and I^\k), and the fact that ui{k — 1) is a 
deterministic function of the information set if' (A;). To evaluate the expected value of x{k — 1) 
given I^\k), we will first change the variables so that we get independent variables. According 
to Proposition 1(c), the innovations yi(k — 1) and y^(k — 1) are independent of y(0 ■.k — 2). 
Thus, 

x?{k) =[A]iE{x(A; - l)|y(0 : A; - 2)} + [A],E{x{k - l)\y,{k - l),Uk - 1)} + B,u,{k - 1) 



-.[A]rx{k - l\k - 2) + Bru^{k - 1) + [A]^E{x{k - l)\y^{k - l),mik - 1)}, 



(8) 



where we used Proposition 4(a) to get the first equality. We will now calculate the last term of 
Equation m. Let St = {1,2,3} and Si = {1,3}. Then 



Cov^ 



yi(^-i) 
Mk-i) 



yi(fc-i) 



E{xik-i)\Mk-i),Mk-i)} 

\ [Uk-l) 

[P(fc-l)]nCf [P{k^l)UC^ 
[P{k - l)]2iCf [P{k - l)UCi 

[P(fc-l)]3lCf [P(fc - 1)]33C3^^ 

= [Pik-lUsAC']lsA[C]sisAPik-l)]s^sdC]ls, + [VWs^] 



yiik-i) 

Mk~l) 



Ci[P{k- 




Ci[P{k- 






-1 


yi{k~ 


1)" 


C3[P{k~ 


l)]3lCf + [^]31 


C3[P{k~ 




h ^33 




yaik- 


1) 



yl(fc-l) 

Uk-l) 



(9) 



where we used Proposition 4(b) to get the first equality and Proposition l(a)-(b) to obtain the 
second equality. Substituting Equation ([9]) into Equation ([8]) shows that x^l\k) is computed as 



x^'\k) =[A]ix{k - l\k - 2) + Biuiik - 1) + K^^l{k-1) i^S(A;-l) 



Mk-l) 
Uk - 1) 



(10) 



where 



Similar results can be obtained for Player 2 and Player 3. Let S2 = {1,2} and 6*3 = {2,3}. 
Then 



x^^\k) =[A]2x{k - l\k - 2) + B2U2ik - 1) + 



411(A:-1) K^2^{k~l) 



x^^\k) =[A]3x{k - l\k - 2) + Bsusik - 1) + K^^i{k - 1) K^Si^ - I) 
where 



yi(fc-l) 
Uk - 1) 



(11) 



(12) 



i^W(fc-l) ifg(fc-l) 



Define the matrix K^^'^ by 





'K^ihk) 





43 (^) 




KSik) 












45 (fc) 


KSik) 



Then equations (|T0|)-(fT2|) can be combined and written in the compact form as 

x^^\k) = Ax{k-l\k-2) + Bu{k-l) + K^^^{k-l){y{k-l)-Cx{k-l\k-2)). (13) 
The Kalman filter iterations for the three-player problem at time k is summarized as follows 



x{k- l|fc-2) 


= Ax{k - 


2\k 


-3)- 


\-Bu{k- 


2)- 


H K{k - 2) {y{k - 2) - Cx{k - 2|fc - 3)) 




= Ax{k - 


l\k 


-2)- 


\-Bu{k- 


1)- 


F K^^\k - l){y{k - 1) - Ci(fc - l|fc - 2)). (14) 



Note that K^^'^ is not the usual Kalman filter gain and that it has a the same sparsity pattern as 
G. Figure |2] shows the overall estimation scheme of Player 1 at time k. 

Remark 1 Both i^I^l and K can be calculated off-line without knowing the control input history 
m(0 : A^- 1). 
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yi(fc- 


2) 




2) 




2) 



r 



Kalman 
Filter 



Viik- 


1) 




1) 



Kalman 
Filter 



x™(k) 



x(k-l|k-2) 



x(k-2|k-3) 



Fig. 2. Optimal estimation scheme of Player 1 at time k. 



C. Estimator properties 

Here we compute some quantities that will help us in the following section. Define 

eW(fc) ■.= x{k)-x^^\k) 
P\k) ■^y{k)-Cx^\k). 

We denote the covariance matrices of e^^\k) and y^^\k) by P^^\k) and Y^^\k), respectively. 

Lemma 1 Let AK{k) — K{k) — K'^^\k). Then the following facts hold: 

(a) PW(A;) = P{k) + AX(fc - l)Y{k - 1) A K^{k - 1). 

(b) yW(fc) = CPW(fc)C'^ + V^. under Assumption 1, Y^^\k) is positive definite. 

(c) P{k) := £ {e^^\k)y^{k - 1)} = Ai^'(A; - l)Y{k - 1). 

Proof: See Appendix. ■ 

IV. Optimal Controller Derivation 

This section shows that finding optimal controller for the three-player problem is equiva- 
lent to solving two separate optimization problems. Before proceeding, we state the following 
proposition. 
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Proposition 2 ( 113]) Define the matrices 

S{k) = A^S{k + I) A + Q,, - {A^S{k + l)B + Q„,) + l)B + Q„„)~' {B^S{k + 1)A + Q^J 

H{k) = B^S{k + l)B + Quu (15) 
L{k) = H-\k){B^S{k + l)A + QL), 

for k = 0, - ■ ■ , N — 1 and where S{N) = Qq. Then the cost function can be written as 

N-1 

J = ^ E I {u{k) - L{k)x{k)fH{k) {u{k) - L{k)x{k)) 

k=0 

V 

J U 

N-1 



Tr{^(0)Po} + + l)W} . 



k=0 

V ' 

W 

Moreover, is independent of the control. 

From Proposition |2} it can be seen that minimizing J is equivalent to minimizing J„. Also, 
under Assumption [2| H{k) is positive definite for all k. 

The first step towards finding the structure of the optimal controller is to decompose the state 
vector into independent terms using the following lemma: 

Lemma 2 The state vector can be decomposed as 

x{k) = x{k) + x(A;), 
where x{k) and x{k) are independent and given by 
x{k) = E{x{k)\y{0 : k - 2)} 

x{k) = eW(A;) + {BF{k - 1) + K^^\k - l))y{k - 1). 

Proof: See appendix. ■ 
Note that the term x{k) is the conditional estimate of the state x{k) given the information 
shared by all players, and x{k) is the estimation error. Now that the state vector has been 
decomposed into independent terms, the control input u{k) can be decomposed in an analogue 
manner. 
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Lemma 3 The control input u{k) can be decomposed into two independent terms 

u{k) = u{k) + u{k), 

where 

u{k) = E{u{k)\y{0 ■.k-2)} 

u{k) = F{k)y^^\k) + F^^\k)y{k - 1), 

and is given by 

fW(A;) = G{k) + F{k)C{K^^\k - 1) + BF{k - 1)). (16) 
Proof: See appendix. ■ 

Remark 2 Since B, C and F are diagonal matrices, G{k) and F{k)CK^^\k — 1) have the 
same sparsity pattern. Similarly, F^^\k) and G{k) have the same sparsity pattern. 

From lemmas [2] and [3} both x{k) and u{k) are functions of y(0 : k — 2) which is independent 
of x{k) and u{k). As a result the cost function can be decomposed as 

N-l 

J„ = ^ E I {u{k) - L{k)x{k)fH{k) {u{k) - L{k)x{k)) 



k=0 

'• v- 

J 



N-l 

+ 



{{"^(k) - L{k)x{k))^ H{k){u{k) - L{k)x{k))y 



k=0 



J 

and the optimal control problem reduces to solving 

Problem 1. minimize J{x, u) 

subject to u{k) is a function of ?/(0 : k — 2). 

Problem 2. minimize J(x, u) 

subject to u{k) = F{k)y^^\k) + F^^\k)y{k - 1), 

F{k) and F^^\k) have specified sparsity structures. 
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The following lemma shows that the optimal solution u{k) for Problem 1 is exactly the optimal 
controller for centralized information structure with two-step delay, where the information set 

of each player is y{0 : k — 2). 

Lemma 4 Suppose assumptions 1 and 2 hold. An optimal solution for Problem 1 is given by 

u{k) = L{k)x{k) 

= L{k)^{x{k)\y{Q:k-2)}. (17) 

Moreover, the optimal value of the cost function J is zero. 

Proof: See appendix. ■ 
We now focus on Problem 2, namely the computation of {F{k)}j^^Q and {-^^^'(^) j^.^]^ 
Recalling the expansions of x{k) and u{k) in terms of y^^\k), e^^\k), and y{k — 1), J can be 
expanded as follows 

N-l 

J = ^ E { (w(fc) - L{k)x{k)f H{k) {u{k) - L{k)x(k)) } 

N-l 

= ^Tr{H{k)F{k)VF'^{k)}+Tr{H{k){F{k)C-L{k))P^^\k){F{k)C-L{k)f} 

fc=0 

+ Tr {ir(A;) (i^^W (A:) - L{k){BF{k - 1) + i^W {k - l))Y{k - (fc) - L{k){BF{k - 1) + if W (A; - 1))^ 

+ 21^- \^H{k){F{k)C - L{k))P{k){F^^^{k) - L{k){BF{k - 1) + K^'^\k - 1))'^} , (18) 

where we used Proposition 4(c). A point worth noticing is that according to Proposition 1 and 
Lemma 1, P^^\ P, and Y are independent of F{k) and F^^^{k). To minimize J with respect to 
F(k) and F^^^{k), we face two difficulties: the first is that F{k) and F^^^{k) must satisfy given 
sparsity constraints; the second difficulty is the existence of coupling terms between F{k — 1) 
and F{k). To overcome these difficulties, we will use the vec operator and the following lemma: 



Lemma 5 Assume that A G M."-^"^ is split into sub-blocks as follows: 



111 



A^ 



A. 



pi 



A, 



pq 
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where A e R"»^"*j far i — 1, . . . ,p and j — 1, . . . ,q. Let S be the set of non-zero sub-blocks of 

A 

S = {A,j\Ai^^Q}, \S\=s. 
Then there always exists a fall column rank matrix E of an appropriate dimension such that 

vec(Aui) 



vec(A) = E 



where Ai^j^^ e S for all k = 1, . . . , s. 

Proof: See appendix. ■ 
The way to construct matrix E is described in Appendix. Lemma 5 ensures the existence of Ei 
and E2 such that 

YCc{F{k)) ^ E^Cl{k), 

vec {F^^\k)) = E^Uk), 

where ^1 and ^2 are vectors formed by stacking all nonzero sub-blocks of F and F^^\ respectively. 
That is, 

vec(F(fc)) =i;i fvec^(Fn) vec^ (F22) vec^CFas)]^, 



6(fe) 



«2(/c) 

We now show how vectorization allows to convert Problem 2 into an unconstrained convex 
optimization problem. 



Lemma 6 Let E = diag{Ei, E2), C{k) 
^i{N -1). Define 



for k = 1,. . . ,N - 1, and ({N) 



Z2{k)=E 



I 
-l0L{k)B I 

I ® L{k)B l] ^ [p'^{k)C^ ® H{k)^ 



yW(A;-l)(^ff(A;-l) 

Y{k-l)(S)H{k) 



I 

~I®L{k)B I 



E, 



I 



E, 



b{k) = E'^ I ^ (cpW(fc - 1) O H{k - 1)) vec(L(fc - 1)) 



E' 



I ® L{k)B l\ ^ (r (fc - 1) H{k)) yQc{L{k)K^^^ {k - 1)) , 
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with 



Zi{N) =£;f (yW(iV- l)^H{N -1))E, 
b{N) ^Ef (cfW(7V - 1) (g) H{N - 1)) wec{L{N - 1)). 



Then Problem 2 is equivalent to 

N-l 

C(i),...,C(JV) ~ 

^ '-T(-\j\ ry I AAA/' ^r^ rT 



r^.,.n = E lc^ik)zAk)ak) + e{k)z^{k)ak + 1) - c^(fc)6(A;) 

i),...,C(A') 2 

C^(iV)Zi(iV)C(A^)-C^(iV)6(iV) (19) 
Moreover, Zi{k) is positive definite for all k. 

Proof: See Appendix. ■ 



fc=i 
1 

2' 



Consider the two time-step case of (19) 



min l^imimi) - emi) + eim{m2) + le{2)z,{2)c{2) - e{2)b{2) . 

C( J>C( )^ ^ ^ ^ ^ ^ 

9i(C(l)) 92(C(1),C(2)) 

(20) 

The optimal C{2) is the one which minimizes g2, i.e. 



C(2) = argmin^72(C(l),C(2)) 

C(2) 

= -Zri(2)(Zj(l)C(l)-6(2)). 



If we substitute the optimal C*(2) into ( [20] ), then we can minimize gi{C,{l)) + (72(C(1), C(2)) 
with respect to C(l)- Therefore, 

= argmin(?i(C(l)) +^72(C(1),C(2)) 
C(i) 

= i?r'(iMi), 

where 

R{1) = Z,{1) - Z,{l)Z^\2)ZUl), 
c(l) = b{l)-Z,{l)Z^\2)b{2). 

The extension to more time steps is straightforward. The result is stated in the following lemma. 
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Lemma 7 Suppose assumptions 1 and 2 hold. Define 

R{k) = Zi{k) - Z2ik)R-\k + l)Zj{k) 
c{k) = b{k) - Z2{k)R-\k + l)c{k + l), 



with the end condition R{N) = Zi(N) and c{N) = b{N). Then optimization problem {19) has 
the unique solution 

^(fc + 1) = + 1) {Z^{k)m - c{k + 1)), (21) 

with initial condition ({1) = i?^^(l)c(l). Moreover, R{k) is positive definite for all k. 

V. Main Results 

We can now state our main result, Theorem 1, which gives the optimal controller for the 
three-player problem. 

Theorem 1 Suppose assumptions 1 and 2 hold. Let x{k) = E{x{k)\y{0 : k — 2)}. Then optimal 
controller for the three-player problem is given by 

u{k) = F{k){y{k) -Cx^^\k)) + F^^\k){y{k - 1) - Cx{k - l\k - 2)) + L{k)x{k), (22) 

where xt^](fc) and x{k — l\k — 2) are the optimal state estimates obtained using the Kalman 



filter iterations ([14]), L is given by Equation (15), and F and F^^l are given by Equation (21). 
Moreover, 

x{k) =x^^\k) - {BF{k - 1) + irW(A; - 1)) {y{k - 1) - Cx{k - l\k - 2)). 
Having derived the optimal controller, a number of remarks are in order. 

Remark 3 A physical interpretation of the optimal control policy is given as follows: The third 
term of optimal controller, L{k)x{k), is exactly the optimal policy for centralized information 
structure with two-step delay, where the information set of each player is y{0 ■.k — 2). The first 
and second terms are correction terms based on local measurements from time k and k ~ 1, 
respectively, which are available to each player. 



Remark 4 The recursive equation {2\_) reveals a new feature present neither in LQG control 
with one-step delay sharing information pattern nor in the state-feedback case: the optimal 
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control gain at time k, ({k), is an affine function of({k — 1). For example, in the state-feedback 
case where yi{k) = Xi{k) for i = 1, 2, 3, we have 

P{k) = E{w{k - l)w'^{k - 2)} = 0. 



According to Lemma 6, Z2{k) = 0, and hence Equation pTj ) reduces to 

C{k) = z^\k)b{k). 



Remark 5 Equating the right hand side of equations (|5]) and (22) shows that the linear function 
f is given by 

f = {L{k) - F{k)C)x{k) - G{k)Cx{k - l\k - 2), 



where G is given by Equation ([16]). Note that both x{k) and x{k — l\k — 2) are linear functions 
ofy{0:k-2). 

Remark 6 If A G M"^", then the optimal controller for the three-player problem has at most 
2n states. 



VI. Numerical Example 

We conclude our discussion of the three-player problem with an example. Consider a simple 
system specified by 





2 





1 




1 










1 








A = 


1 


2 





, B = 





1 





, c = 





1 










1 


2 










1 










1 



w and V are Gaussian with zero mean and identity covariance matrix. The time horizon is 
chosen to be 1000 and the cost weight matrices are given by 
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and Qo = Q^^. 

We will compare the optimal controller for the three-player problem to controllers for the 
following information structures 
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Fig. 3. The graph illustrates the communication structure of one-step delay information pattern. Each controller passes 
information to both neighbors after one-step delay. 

1) Centralized with two-step delay: Ui{k) = iJ,i[y{0 : A; — 2)), 

2) One-step delay sharing information pattern: Ui{k) = ^i(^yi{k),y{0 : A; — 1)), 

3) Centralized without delay: Ui{k) = /Uj(?/(0 : k)). 

The one-step delay sharing information pattern studied in Q-Q is specified by the graph in 
Figure [3j 

By minimizing cost function (|4]), we obtain Table 1. Centralized controller without delay has 
the lowest cost as expected. The three -player controller outperforms the centralized controller 
with two-step delay by a substantial margin, and only around 1.74% higher than one-step delay 
sharing information pattern control. In other words, for three-player problem, there is a slight 
benefit of having two-way communication between controllers. Comparison of the costs shows 



TABLE I 

Simulation Results for Total Cost 



Control law 


Cost mean 


Centralized with delay 


14757 


Three-player 


339.9 


One-step delay information pattern 


334.1 


Centralized without delay 


188.8 



the benefits of using all available information. 

VIL Conclusion 

In this paper, we presented an explicit solution for a distributed LQG problem in which three 
players communicate their information with delays. This was accomplished via decomposition of 
the state and input vectors into two independent terms and using this decomposition to separate 
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the optimal control problem to two subproblems. Computing the gains of the optimal controller 
requires solving one standard discrete-time Riccati equation and one recursive equation. Future 
work will continue to extend our approach to the infinite-horizon and more general networks. 
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VIII. Appendix 

A. Preliminaries 

Proposition 3 ( / [J?] / ) If A, B, C, D, X and Y are suitably dimensioned matrices, then 

(a) vQc{AXB) = (5^ ® A)wec{X), 

(b) If A and B are positive definite, then so is A® B, 

(c) Tr{AXBY^} = vec^(r)(5^ ® A)vec(X), 

(d) {A(^B)-^ = A-^^B-\ 
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(e) Let X G M™^", then there exists a unique permutation matrix Pm,n ^ M™"^™" such that 
vec(X^) = Pm,nVec(X). The matrix Pm,n is given by 

m n 
1=1 j=l 

where Eij G M™^" has a one in the entry and every other entry is zero. 

Proposition 4 [13]) Let x, y and z be zero-mean random vectors with a jointly Gaussian 
distribution, and let y and z be independent. Also, let S be a symmetric matrix. Then the 
following facts hold: 

(a) 'E{x\y,z] = Y^{x\y] + ¥j{x\z]. 

(b) E{x|?/} = Cov{s,?/}Cov"^{y,y}?/. 

(c) ^{x'^Sx} = Tr{5Cov{x,x}}. 

(d) E{x|?/} and x — E{x|?/} are independent. 

B. Proof Lemma 1 

To express the conditional estimate x{k\k — 1) in terms of x^^\k), we substitute Equation ( fT3| ) 
into Equation (|6]) to eliminate Ax{k\k — 1) + Bu{k). We have 

x{k\k - 1) = sW(A;) + {K{k - 1) - K^^\k - 1)) {y{k - 1) - Cx{k - l\k - 2)) 

= x^^\k) + AK{k-l)y{k-l). (23) 

Plugging x{k\k — 1) = x{k) — e{k) and x^^^{k) = x{k) — e^^\k) into Equation ([23]) leads to 

e^^\k)=e{k) + /\K{k-l)y{k-l). (24) 



Since e{k) is independent of y(0 : A; — 1), the two terms on the right hand side of Equation ( |24| ) 
are independent. Thus, 

PW(A;) =E|eW(A;)eW(A;)^| 

= P{k) + AK{k - l)Y{k - 1) A K'^{k - 1), 
yW(fc) =E|yW(A;)yW(A;)^} 

= CP^^\k)C^ + v, 
P(A;) =E{eW(A;)y^(A;- 1)} 

= /\K{k-l)Y{k-l). 
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C. Proof Lemma 2 

The independence between x{k) — x{k) and x{k) can be established by Proposition 4(d). To 
calculate x{k), we proceed in three steps. First consider 

u{k - 1) = F{k - l)y{k - 1) + G{k - l)y{k - 2) + f{y{0 : A; - 3)), 

where we used Equation ([5]). Since G{k — l)y{k — 2) + /(y(0 : A; — 3)) is a deterministic function 
of ?/(0 : k — 2), we have 

u{k-l) - E{u{k - l)\y{0 : k - 2)} = F{k-l){y{k - 1) ~E{y{k - l)\y{0 : A; -2)}) 

= F{k-l)y{k-l), (25) 

where we used the definition of y (Equation Q) to get the second equality. Second, consider 

xW(A;) = Ax{k - 1|A; - 2) + Bu{k - 1) + irW(A; - l)y{k - 1), 
where we used Equation ( [T3| ). Since — 1|A; — 2) is a linear function of y{0 : A; — 2), we have 

jW(fc) -E{jW(fc)|?/(0 : fc - 2)} = K^^Hk-l)y{k-l) + B{u{k-l)-E{u{k-l)\y{0 : k - 2)}) 

= (is:W(fc - 1) + BF{k - l))y{k - 1), (26) 

where we used the independence of y{k — 1) and y{0 : A; — 2) to get the first equality, and 
Equation ( [25] ) to obtain the second equahty. Finally, note that x{k) = e^^^(k) + x^^^{k). Thus, 

x{k) = x{k) - E{x{k)\y{0 : k - 2)} 

= eW(A:) + (JW(A;) - E{xW(A;)|y(0 : k - 2)}) 

= eW(A;) + (kW(A; - 1) + BF{k - 1)) y(A; - 1), (27) 
where we used the independence of e^^^{k) and y{0 : A; — 2) to get the second equality and 



Equation (26) to obtain the last equality. 



D. Proof Lemma 3 

According to Proposition 4(d), u{k) is independent of u{k) — u{k). Note that v{k) is inde- 
pendent of the previous outputs, so 

y{k) - E{y{k)\y{f) ■.k-2)}= v{k) + C{x{k) - E{x{k)\y{0 : k - 2)}) 

= v{k) + C(eW(A;) + {BF{k - 1) + K^^\k - 1)) y{k - 1)) 

= yW(A;) + C{BF{k - 1) + irW(A; - l))y{k - 1), (28) 
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where we used Equation (27) to get the second equahty and the definition of yl^^ (EquationQ) 
to obtain the last equality. Since f{y{0 : /c — 2)) is a linear function of y{0 : k — 2), we have 

u{k) =u{k) - E{u{k)\y{0 : k - 2)} 

=F{k) [y{k) - E{y(fc)|y(0 : A; - 2)}) + G{k) [y{k - 1) - E{y(A; - l)|y(0 : A; - 2)}) 
=F{k) {y^^\k) + C{BF{k - 1) + K^^\k - l))y{k - 1)) + G{k)y{k - 1) 
=F{k)y^^\k) + {F{k)C{BF{k - 1) + ^^(A; - 1)) + G{k))y{k - 1), 



where we used Equation ( [28] ) and the definition of y (Equation (|7])) to get the third equality. The 
proof is completed by defining 

fW(A;) = G{k) + F{k)G{K^^\k - 1) + BF{k - 1)). 



E. Proof Lemma 4 

Due to the assumptions, H{k) is positive definite and hence all terms in the J are positive. 



Since u{k) and x{k) are functions of y(0 : k — 2), the optimal controller is given by (17). 



F. Proof Lemma 5 

Let Aj G ]R"^'^J denote the j*^ block column of matrix A. According to Proposition 3(e), we 
have 



v&c{Aj) = vec 



A 



\ 



J 



/ilj . . . /ipj 



P 

^ rnj,n 



vec(A^; 



Pni,m,Vec(A 



-Pnp , m j vec ( v4p j 

vec(y4ij 



-Pmj,ndi3§(-Prii,Tnj ) • • • ; -^rip,'. 



vec (A 
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Let Pj = Pm^.,„diag(P„i,^., . . . , Pn^,m,)- Then 



vec(A) 



vec(Ai) 



vec(Ag) 



diag(Pi,...,P, 



P 



vec(y4 



vec(A 



pi J 



vec(y4 



vec(A 



(29) 



Note that vector a a consists of all pq sub-vectors vec(Aii), . . . ,vec(y4pg). Let a\ denote the 
vector containing only nonzero sub-vectors of a^- We define A = {i\[aA]i 7^ 0}. Let Tj = 

T 











be the block matrix with an identity in the i^^ block row. It is easy to see 



that there exists full column rank matrix T whose columns are Tj for j E A such that a a = Tua- 



This implies that Equation ([29]) can be written as 

vec(A) = PTqa. 
The proof is completed by defining E = PT. 

G. Proof Lemma 6 

The equivalence of optimization problems follows simply by using the vec operator. First note 



that vec(P(A;)) = Eiii{k) 



I 



EC{k + 1). Using Proposition 3(c), the first term on the 



right-hand side of ( fT8| ) can be written as 

Tr {H{k)F{k)VF{kf] = vec^(P(fc)) {V ® H{k))vec{F{k)) 



e{k + l)E' 



I 



{V®H{k)) 



I 



EC{k + l). 



(30) 



The second term on the right-hand side of ( [181 ) can be written as 

Tr|i?(fc)(F(fc)C- L(/c))pW(/c)(i^(fc)C- L(fc))^| ^ vec^ {Fik)){CP^^^{k)C^ (g) H{k))wec{F{k)) 



2vec'^ (F(fc)) (CPl^l (fc) ® H{k))vec {L{k)) 



vec 



(L(A:))(PW(A:) (»i?(fc))vec(L(fc)) 



e{k + l)E^ 
-2(^{k + l)E'^ 



I 



/ 



{CP^^\k)C'^ ®H{k)) 



I 



EC{k + l) 



(CpW(fc) ® i/(/c))vec(L(fc)) 



(L(/c)) (PW (/fc) (g) i7 (fc)) vec(L(fc)) . 

(31) 
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Likewise, vec(FW(A;)) = E2^2{k) =01 EC{k) and 

vec (FW(fc) - L(/c)(ifW(fc - 1) + BF{k - 1))) 

= vec(FW(fc)) - (/(g)i(fc)B)vec(F(fc- 1)) - vec(i(fc)i^W (/c - 1)) 
-/ (g) L{k)B /] £;C(fc) - vec(L(A:)ifW (fc - 1)) , 
where we used Proposition 3(a) to obtain the second equality. The third term on the right-hand 



side of (fT8l) can be written as 
Tr 



(FW(fc) - - 1) + BF{k - 1))) Y{k - 1) (F^^^{k) - L(k){K^^\k - 1) + BF{k - 1)))" 



C^{k)E' 



-I(g)L{k)B I 



{Y{k-l)(g) H{k)) 



-I(g)L{k)B I 



Em 



2C{k)E'^ ~I®L{k)B I {Y{k-l)®H{k))v&c{L{k)K^^\k^l)) 
vec^(L(fc)xW(fc - 1)) (f (fc - 1) (g) i7(fc))vec(i(fc)i^W(/c - 1)). 



(32) 



The last term on the right-hand side of (18) can be written as 



Tr 



H{k) {F{k)C - L{k)) P{k) - L{k){K^^\k - 1) + BF{k - 1)) 



-e{k)E^ 



-I(g)L{k)B I 



I 



i?C(fc + i) 



-I(g)L{k)B I 



{P'^ {k) (g) H{k))vec{L{k)) 



e{k+l)E^ 



I 



(CP(fc) ® H{k))wtc{L{k)K^^\k ~ 1)) 



+ vec^ {L{k)K^^\k - 1)) (P^(fc) (g) H{k))vec{L{k)). 



(33) 



Substituting (|30])-(l33]) back into ([18]), noting that ^[^'(A;) = CP^^^{k)C^ + V, and omitting 
constant terms we arrive at ( fTO] ). 

The proof can be completed by showing that Zi{k) is positive definite. Since Y^^^{k), Y{k), 
and H{k) are positive definite according to assumptions 1 and 2, Y^^^{k — 1) //(A; — 1) and 
Y{k — 1) ®H{k) are positive definite according to Proposition 3(b). Therefore, Since E has full 
column rank, Zi{k) is positive definite. 



H. Proof Lemma 7 

To prove the theorem, we start from the endpoint and iterate backwards in time. Define 

n(7V) = min(ic^(7V)Zi(iV)C(Af) + C^(A^ - \)Z^{N - l)C(iV) - C {N)h{N) 



C(JV) [2 



(34) 
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Since Zi{N) is positive definite, by taking derivative with respect to C(^)' the optimal value of 
((N) is given by 

C{N) = -R-\N){Z^{N - l)C(iV - 1) - c(iV)), 
where R{N) = Zi{N) and c{N) = b{N). 



By substituting the optimal value of C(^) into Equation ( [34] ), we have 

n(7V) = - ic^(7V - 1)Z2{N - l)R-'iN)Z^iN - l)aN - 1) 

+ C^(^ - 1)Z2{N ~ 1)R-\N)c{N) - ^c^{N)R-\N)c{N). 

Note that the last term is constant and independent of Ci^ ~ !)• 
We proceed similarly and define 

n(iV-l)= min \lc^{N -l)Zi{N -l)C{N -l) + C'^{N -2)Z2{N -2)CiN -1) 

- C{N - l)b{N - 1) + n(A^) 
= min ll(:'^{N-l)R{N-l)C{N-l) + (:'^{N^2)Z2{N~2)C{N-l)-C'^{N-l)c{N-l)\, (35) 

where 

R{N - 1) = Zi{N - 1) - Z2{N - 1)R-\N)Z^{N - 1), 
c{N - 1) = b{N - 1) - Z2iN - l)R-\N)c{N - 1). 



Equation (35) is the same as Equation (34), but with the time arguments shifted one step. Thus, 



^*(iV - 1) = -R-\N - 1) {ZjiN - 2)C(iV - 2) - c(iV - 1)) . 
The procedure can now be repeated, and 

n(l)=min|^C^(l)i?(l)C(l)-r(l)c(i: 

Therefore, 

C{l) = R-\l)c{l). 



